Add C++ runtime for spleeter about source separation #2242

csukuangfj · 2025-05-23T14:29:35Z

Usage

Build sherpa-onnx and download model files

git clone https://github.com/k2-fsa/sherpa-onnx
cd sherpa-onnx
mkdir build
cd build
cmake ..
make

# go to https://github.com/k2-fsa/sherpa-onnx/releases/tag/source-separation-models

wget https://github.com/k2-fsa/sherpa-onnx/releases/download/source-separation-models/sherpa-onnx-spleeter-2stems-fp16.tar.bz2

tar xvf sherpa-onnx-spleeter-2stems-fp16.tar.bz2

wget https://github.com/k2-fsa/sherpa-onnx/releases/download/source-separation-models/qi-feng-le-zh.wav
wget https://github.com/k2-fsa/sherpa-onnx/releases/download/source-separation-models/audio_example.wav

Run it with audio_example.wav

./bin/sherpa-onnx-offline-source-separation \
  --spleeter-vocals=sherpa-onnx-spleeter-2stems-fp16/vocals.fp16.onnx \
  --spleeter-accompaniment=sherpa-onnx-spleeter-2stems-fp16/accompaniment.fp16.onnx \
  --num-threads=2 \
  --debug=0 \
  --input-wav=audio_example.wav \
  --output-vocals-wav=output_vocals.wav \
  --output-accompaniment-wav=output_accompaniment.wav

The output logs are

/Users/fangjun/open-source/sherpa-onnx/sherpa-onnx/csrc/parse-options.cc:Read:372 ./build/bin/sherpa-onnx-offline-source-separation --spleeter-vocals=sherpa-onnx-spleeter-2stems-fp16/vocals.fp16.onnx --spleeter-accompaniment=sherpa-onnx-spleeter-2stems-fp16/accompaniment.fp16.onnx --num-threads=2 --debug=0 --input-wav=audio_example.wav --output-vocals-wav=output_vocals.wav --output-accompaniment-wav=output_accompaniment.wav

OfflineSourceSeparationConfig(model=OfflineSourceSeparationModelConfig(spleeter=OfflineSourceSeparationSpleeterModelConfig(vocals="sherpa-onnx-spleeter-2stems-fp16/vocals.fp16.onnx", accompaniment="sherpa-onnx-spleeter-2stems-fp16/accompaniment.fp16.onnx"), num_threads=2, debug=False, provider="cpu"))
Started
Done
Saved to write to 'output_vocals.wav' and 'output_accompaniment.wav'
num threads: 2
Elapsed seconds: 0.469 s
Real time factor (RTF): 0.469 / 10.919 = 0.043

Note: It runs on my macOS (x64) with CPU.

audio_example.wav

audio_example.mov

output_vocals.wav for audio_example.wav

output_vocals.mov

output_accompaniment.wav for audio_example.wav

output_accompaniment.mov

Run it with qi-fen-le-zh.wav

./bin/sherpa-onnx-offline-source-separation \
  --spleeter-vocals=sherpa-onnx-spleeter-2stems-fp16/vocals.fp16.onnx \
  --spleeter-accompaniment=sherpa-onnx-spleeter-2stems-fp16/accompaniment.fp16.onnx \
  --num-threads=2 \
  --debug=0 \
  --input-wav=./qi-feng-le-zh.wav \
  --output-vocals-wav=output_vocals.wav \
  --output-accompaniment-wav=output_accompaniment.wav

Output logs are

/Users/fangjun/open-source/sherpa-onnx/sherpa-onnx/csrc/parse-options.cc:Read:372 ./build/bin/sherpa-onnx-offline-source-separation --spleeter-vocals=sherpa-onnx-spleeter-2stems-fp16/vocals.fp16.onnx --spleeter-accompaniment=sherpa-onnx-spleeter-2stems-fp16/accompaniment.fp16.onnx --num-threads=2 --debug=0 --input-wav=./qi-feng-le-zh.wav --output-vocals-wav=output_vocals.wav --output-accompaniment-wav=output_accompaniment.wav

OfflineSourceSeparationConfig(model=OfflineSourceSeparationModelConfig(spleeter=OfflineSourceSeparationSpleeterModelConfig(vocals="sherpa-onnx-spleeter-2stems-fp16/vocals.fp16.onnx", accompaniment="sherpa-onnx-spleeter-2stems-fp16/accompaniment.fp16.onnx"), num_threads=2, debug=False, provider="cpu"))
Started
Done
Saved to write to 'output_vocals.wav' and 'output_accompaniment.wav'
num threads: 2
Elapsed seconds: 1.262 s
Real time factor (RTF): 1.262 / 26.102 = 0.048

qi-fen-le-zh.wav

qi-feng-le-zh.mov

vocals for qi-fen-le-zh.wav

output_vocals.mov

accompaniment for qi-fen-le-zh.wav

output_accompaniment.mov

Fixes #2235

CC @acely

acely · 2025-05-23T16:07:48Z

太效率了，必须致敬一下！
不过说回来，spleeter的效果并不理想，他最初是为了「去除人声」且获得伴奏而研发的，所以输出的伴奏要比人声部分质量好，人声听起来很糊。
所以说我还是建议试试mdx-net模型，毕竟他那个Voc_FT是专门为了「提取人声」而训练的。

acely · 2025-05-23T16:57:24Z

我用Voc_FT提取了上面两段音频，效果可以对比下。

1_main.mov

2_main.mov

csukuangfj · 2025-05-23T22:34:58Z

太效率了，必须致敬一下！
不过说回来，spleeter的效果并不理想，他最初是为了「去除人声」且获得伴奏而研发的，所以输出的伴奏要比人声部分质量好，人声听起来很糊。
所以说我还是建议试试mdx-net模型，毕竟他那个Voc_FT是专门为了「提取人声」而训练的。

请问你说的这个模型，在cpu上的速度如何？能否测试下RTF?

acely · 2025-05-24T02:43:00Z

我的测试机器是Macmini M4Pro芯片，用Java调用onnxruntime加载的。
测试素材为45分钟的音频：
-纯CPU耗时850秒，RTF=0.3148
-开启CoreML支持耗时350秒，RTF=0.13

dfengpo · 2025-05-25T14:26:22Z

mdx-net

你这个效果太惊艳了，请问哪里可以下载mdx-net模型的onnx版本，我非常需要这个模型。顺便问下比如商场这种人流噪音下的录音也一样可以处理吗？

acely · 2025-05-25T15:42:53Z

参考https://github.com/seanghay/uvr-mdx-infer
模型在https://github.com/TRvlvr/model_repo/releases/tag/all_public_uvr_models
找到我用的这个UVR-MDX-NET-Voc_FT.onnx

acely · 2025-05-25T15:44:15Z

@dfengpo 降噪我不确定，手里没有相应的测试素材

dfengpo · 2025-05-29T08:16:06Z

@acely

请问下你上面的分离示例，是使用以下脚本跑出来的结果吗？
https://github.com/acely/uvr-mdx-infer/blob/main/separate.py
我看源作者说他有更简单的实现：
https://github.com/seanghay/vocal/blob/main/vocal/__init__.py
想确认一下你是用哪个脚本跑出来的结果，非常感谢

acely · 2025-05-29T16:36:01Z

@acely

请问下你上面的分离示例，是使用以下脚本跑出来的结果吗？ https://github.com/acely/uvr-mdx-infer/blob/main/separate.py 我看源作者说他有更简单的实现： https://github.com/seanghay/vocal/blob/main/vocal/__init__.py 想确认一下你是用哪个脚本跑出来的结果，非常感谢

用的这个https://github.com/acely/uvr-mdx-infer/blob/main/separate.py

csukuangfj added 15 commits May 22, 2025 17:05

begin to add files

4208b68

Merge remote-tracking branch 'dan/master' into spleeter-cpp

1d72a48

add source separation

4c88a6a

add model config

e50fe9e

begin to add impl

13dc9b2

begin to add spleeter impl

6250c55

Fix wasm for speech enhancement

5bd106a

Change input shape to make it easier for C++

4d5c45e

Support reading multi-channel wave files

1ead04c

Test read and write multi-channel waves

6fa3854

Begin to implement processing

11cf8ea

add meta data

b36bf86

Add spleeter model

8d9e495

first working version

76578e9

add resampling

9dba343

csukuangfj merged commit 716ba83 into k2-fsa:master May 23, 2025
184 of 218 checks passed

csukuangfj deleted the spleeter-cpp branch May 23, 2025 14:31

This was referenced May 31, 2025

Add UVR models for source separation. #2266

Merged

Add C++ support for UVR models #2269

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add C++ runtime for spleeter about source separation #2242

Add C++ runtime for spleeter about source separation #2242

Uh oh!

csukuangfj commented May 23, 2025

Uh oh!

Uh oh!

acely commented May 23, 2025

Uh oh!

acely commented May 23, 2025

Uh oh!

csukuangfj commented May 23, 2025

Uh oh!

acely commented May 24, 2025

Uh oh!

dfengpo commented May 25, 2025

Uh oh!

acely commented May 25, 2025

Uh oh!

acely commented May 25, 2025

Uh oh!

dfengpo commented May 29, 2025 •

edited

Loading

Uh oh!

acely commented May 29, 2025

Uh oh!

Uh oh!

Add C++ runtime for spleeter about source separation #2242

Add C++ runtime for spleeter about source separation #2242

Uh oh!

Conversation

csukuangfj commented May 23, 2025

Usage

Build sherpa-onnx and download model files

Run it with audio_example.wav

audio_example.wav

output_vocals.wav for audio_example.wav

output_accompaniment.wav for audio_example.wav

Run it with qi-fen-le-zh.wav

qi-fen-le-zh.wav

vocals for qi-fen-le-zh.wav

accompaniment for qi-fen-le-zh.wav

Uh oh!

Uh oh!

acely commented May 23, 2025

Uh oh!

acely commented May 23, 2025

Uh oh!

csukuangfj commented May 23, 2025

Uh oh!

acely commented May 24, 2025

Uh oh!

dfengpo commented May 25, 2025

Uh oh!

acely commented May 25, 2025

Uh oh!

acely commented May 25, 2025

Uh oh!

dfengpo commented May 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

acely commented May 29, 2025

Uh oh!

Uh oh!

dfengpo commented May 29, 2025 •

edited

Loading